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In this paper we present a model of collaborative evaluation that has been used to engage teachers in data-based deci¬ 
sion making for improving teaching and learning in mathematics and science. We examine three external challenges that 
threaten the process of continuous school improvement; namely, making sense of data, policy changes, and curriculum 
changes. In addition, we describe how the collaborative evaluation model facilitated progress beyond these challenges. 

Keywords: data-based decision making, evaluation, mathematics education, science education 



< 

m 


i A 
in 
C 


A cross the country, administrators are using various methods for collaboration and teaming to help teachers 
improve their professional work. Professional Learning Communities (PLCs) are functioning in many schools, 
with attention to characteristics such as supportive and shared leadership, shared values and norms, collective 
learning and application of learning, supportive conditions, and shared practice (Hord, 2004).The PLCs have typically 
been established to embody the principles of ensuring that students learn, developing a culture of collaboration, and 
focusing on results (DuFour, 2004; DuFour & Eaker, 2002; DuFour & Eaker, 1998). Concurrently, data-based decision 
making has become a common practice for school administrators engaged in school improvement processes with 
the intent of improving teaching and learning. 

It is imperative that principals also employ processes to help teachers examine various types of data and think about 
how data can be used to inform instructional decisions that lead to increased student achievement. In the advent 
of the accountability movement it makes sense to have teachers take a more proactive role with data. Using data 
diagnostically to inform decisions about how to improve student achievement is an effective method for helping 
teachers change instruction and impact student learning positively (Englert, K., Fries, D., Martin-Glenn, & Douglas, 
2007; Kowlaski, Lasley & Mahoney, 2008). As a result, there is a growing need for administrators to provide leadership 
in data analysis and data use for the purposes of making informed educational decisions. 

The process of engaging teachers in data analysis is potentially fraught with challenges. School administrators are 
well aware of the internal hindrances they might face motivating staff and finding time for teachers to meet as a group 
and examine data. However, there are also external forces that administrators will face when they engage teachers 
in the process of using data to inform decisions. It is important for educational leaders to be aware of external forces 
so they can help teachers properly prepare for data-based decision making and overcome the challenges that can 
thwart progress. 
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The purpose of this study was to identify external challenges that urban schools might face when using student data 
to make decisions and to analyze how a collaborative evaluation model for data-based decision making might help 
administrators confront the external challenges identified. In this paper we describe a collaborative evaluation model 
that we developed with funding by the National Science Foundation (NSF). The model uses an innovative immersion 
approach to data-based decision making combined with the concept of PLCs that helps school administrators and 
teachers improve school practice (Huffman, Thomas, & Lawrenz, 2008). We used the collaborative evaluation model 
in schools to overcome external forces that hindered progress toward improved practices. 
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School administrators and teachers are faced with an array of tests used to judge annual yearly progress (AYP). In 
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many ways schools are bombarded with data. State tests, district tests, national tests, and classroom tests are all used 
to gather information about student achievement. This can lead to a situation in which schools are overwhelmed 
with information about students. There are so many different tests and sources of data that it is sometimes difficult 
to figure out how to react in such a way that helps students improve. Sometimes teachers will meet with one another 
to discuss data, but typically teachers are on their own to sort through the information and figure out how best to 
respond. 

Some schools have moved towards collaboration to help teachers work with data and develop a data-based response. 
Collaborations that engage teachers in inquiry, reflection, and data-based decision making have all been shown to 
be powerful tools for influencing an individual's beliefs and theories of teaching and learning (Bissex, 1994; Cochran- 
Smith & Lytle, 1993,1999; Huffman & Kalnin, 2003; Huffman &Thomas, 2009). Participating in collaborative inquiries 
can help teachers become what Kowalski, Lasley, and Mahoney (2008) describe as "evidenced-based decision makers" 
(p. 256). They contend that evidence-based decision makers not only know how to implement decisions, but also 
know how to examine school and classroom environments in a broader sense. 

Helping a teacher become an 'evidence-based decision maker' involves more than just looking at the student test 
scores from ones'own classroom. It involves school administrators helping teachers look more broadly beyond the 
classroom and attempting to think about the school and district as a whole. This is a new role for many teachers. 
Teachers have historically focused on their own classroom and administrators have focused on the school. The 
movement towards creating 'evidence-based decision makers' in schools will require everyone to think more broadly 
about roles and to consider factors outside the classroom that may affect student achievement and ultimately the 
schools'Annual Yearly Progress (AYP). 

THE COLLABORATIVE EVALUATION COMMUNITIES PROJECT 

We received funding from the NSF to develop the Collaborative Evaluation Communities Project (CEC).The purpose 
of the project was to help teachers in urban schools build assessment and evaluation capacity by engaging in 
long-term, collaborative inquiry related to mathematics and science education. Stockdill, Baizerman, and Compton 
(2002) defined evaluation capacity building (ECB) as "the intentional work to continuously create and sustain 
overall organizational processes that make quality evaluation and its use routine" (p. 14). With the current trend of 
organizations moving toward internal evaluations to improve practices (Torres & Preskill, 2001), finding ways to help 
schools build evaluation capacity is important. The CEC project provided a unique structure for immersing teachers 
in a continual evaluation process in an attempt to improve teachers' capacity to use data and engage in continuous 
improvement through data-based decision making. 

The CEC project provided long-term collaborative experiences for school administrators and teachers as a means of 
developing the evaluation capacity of K-8 schools. The collaborative experiences occurred through the formation 
of collaborative evaluation communities (CEC teams) consisting of teachers, school administrators and university 
faculty who had expertise in both evaluation and mathematics and science education. The key concept behind the 
CEC project was that by immersing teachers in the evaluation process we could help build evaluation capacity and 
bridge the gap between school district evaluation efforts and the teaching and learning of science and mathematics. 

The Collaborative Evaluation Communities Process 

The CEC project used an inquiry cycle developed by the National Research Council (NRC, 1999) to engage the CEC 
teams in the ongoing process of evaluation (see Figure 1).The professional development program developed by NRC 
is called "Global Decision Making Through Local Action: Using TIMSS to Make Data-based Decision." It was developed 
to help school personnel examine student achievement data and to make decisions based on data.The model utilizes 
a cycle that begins with an examination of student achievement data at the national, state, and local levels. At this 
stage, the CEC teams explored data and considered how those data might inform the evaluation of mathematics and 
science programs in the school.The CEC teams then proceeded through an iterative process of evaluation. By the end 
of the project, the process established a continuous improvement cycle (see Figure 1) that helped the collaborative 




95 


Thomas and Huffman 















t 


evaluation communities use evaluation as a tool to create change in their schools and empowered the teachers to 
become leaders in their schools (Huffman and Thomas, 2009). 
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Figure J. Inquiry Cycle 

The following description of one school's work illustrates how the process can be implemented. The first meetings 
of the CEC team at the school were dedicated to exploring national, state, and district student achievement data in 
mathematics. Next, the team related those data to the context of the school by considering the characteristics of 
their students, the curriculum, and their own instructional strengths and weaknesses. The team generated several 
areas that could provide a focus for evaluation. The broad spectrum of issues was condensed by the team into the 
following four categories and subcategories: 

1. Professional Development 
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a. Teacher training related to district adopted curriculum 

b. Curriculum implementation 

c. Creating lessons to motivate students to be active participants in their own learning and encourage them to 
value mathematics 

2. Curriculum Alignment 

a. Curriculum alignment with standards and assessments 

b. Transference of knowledge 



i. from curriculum—state assessments 

ii. from Reading—Math (ex. Venn Diagrams) 

iii. Problem solving ability between subjects 
3. Instruction 

a. Instructional practices to include higher level thinking strategies for students and to examine teachers' 
current practices 

b. Accommodating English as a second language learners and learners who struggle to learn mathematics 
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c. Creating lessons to motivate students to be active participants in their own learning and encourage them to 
value mathematics 
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4. Assessment 

a. How to measure student progress 

b. Develop alternative assessments to the formative assessments offered by the state and to pencil and paper 
assessments 

Although several evaluation questions were generated related to the categories, the team selected one question to 
guide the group's first evaluation activities. The team decided to focus on addressing the question, "How is student 
learning affected when teachers study the lessons and complete the investigations in the curriculum themselves 
before teaching?" All CECteam members agreed that this question would emphasize instruction and be more focused 
than the broad categories mentioned above. The team also believed this question addressed both student learning 
and professional development. 

An evaluation process was established to help the group gather data related to the question, through which the CEC 
team 


• participated in collaborative study of mathematics lessons; 

• measured student achievement Pre/Post and at intervals during the school year; 

• collected student-derived lesson data: observations, interviews, journals, and work samples; 

• collected teacher derived lesson data: observations, teacher reflections; and 

• used quantitative and qualitative methods for analysis. 

The CEC team created a protocol for collaborative study of mathematics lessons for each unit in the curriculum. To 
facilitate data collection, the group modified/produced instruments to gather student data, lesson observation data, 
teachers'lesson analysis data, and student lesson reflection data. 

There were several outcomes of the team's collaborative evaluation efforts. The team decided to analyze student 
needs based on student assessment data from the state mathematics assessment and the Northwest Evaluation 
Association, Measures of Academic Progress (MAP) Test. The results of those analyses informed decisions about 
mathematics teaching and learning in the classroom. While this may seem like professional work that teachers 
should already have been engaged in together, it was not common practice at the school.The group also worked to 
generate common pre/post tests that could be used for all units to measure student progress.The need for pre/post 
test agreement was highlighted by the teachers. Establishing common tests helped the teachers make observations 
about student learning which lead to instructional decisions that were connected to student data. The team used 
assessment data to conduct a curriculum alignment study. Analyzing student assessment results and expectations 
from the state mathematics standards, the team aligned mathematics units/lessons with the standards and student 
needs. These types of collaborative professional activities were not part of teachers'work prior to participation in the 
CEC project. Through these activities the teachers were not only using external data to make decisions, they were also 
gathering their own data. 


METHODS 


The emergent and cyclical nature of qualitative research often leads to new insights and questions (Ridenour & 
Newman, 2008). This was the case throughout our work on the CEC project. After engaging as participant observers 
during the four years of the project and analyzing the data in pursuit of other research questions, we began to notice 
that the challenges administrators face when involving teachers in data-based decision making sometimes extended 
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beyond the school. From this observation two questions guided this study: 

• First, what external challenges do school administrators face when helping teachers engage in data-based 
decision making? 

• Second, how does a collaborative evaluation model help administrators confront the external challenges 
identified? 

To address these two questions we reviewed data from the project log gathered throughout the CEC project. Our 
notion of the project log is aligned with Ely's and colleagues'(1991) view of the log functioning as "the repository for 
all the data that have been gathered"that represents a "cohesive history" of the project (Ely, Anzul, Friedman, Garner, 
Steinmetz, p. 69). First, we conducted a critical incidence analysis of the data from the project log to extract evidence 
that characterized challenges faced by the participants. Next, data related to the incidences that were classified 
as representing external challenges were analyzed to develop codes that captured the essence of each incidence. 
Theses codes were then analyzed as we considered ways to collapse and integrate ideas that were in turn organized 
into common themes, which depicted the hurdles administrators needed to overcome. When we systematically 
reanalyzed the data using this thematic approach our analysis of field notes, focus group transcripts, and interviews 
lead us to identify three hurdles schools may face when interested in data-based decision making. We also identified 
how the collaborative evaluation process helped the CEC teams move beyond those challenges. 

Data Sources 

The project log included participant observer field notes, focus group transcripts and interviews with CEC participants 
during and after participation in the project.The field notes were recorded at monthly CEC work sessions during the 
school year (9 per year) and a CEC team summer workday for each team across four years of the project. Transcripts 
from two focus group sessions with a CEC team (one each at two of the three elementary schools) were also used in 
this study. The focus group sessions were conducted to gain an overall sense of the collaborative evaluation process 
from two schools that were at the end of their formal participation in the project. The interviews analyzed for this paper 
included six semi-structured interviews with CEC team teachers and one unstructured interview conducted jointly 
with a CEC team principal and a CEC team instructional coach.The teacher interviews were recorded and transcribed. 
At the request of the principal and instructional coach, the educational leader interview was not recorded, but the 
interviewer wrote detailed notes of the interview, including direct quotes when possible. All interview transcripts or 
notes were reviewed and approved by the participants. 

Participants 

The CEC project included collaborative teams at three urban elementary schools that served students in Kindergarten 
through 5th grade. The first school involved in the project was located in an industrial section of a large mid-west city. 
The neighborhood was depressed economically, with housing that had not been well maintained. The school was 
crowded, with several grades taught in auxiliary buildings off to the side of the main building. Altogether there were 
approximately 300 students in the school. Of those, 40% of the students were of Hispanic origin and 50% were black. 
Large numbers of the students were learning English as a Second Language (ESL), and the teachers indicated that 
reading proficiency generally was at a low level. The CEC team consisted of 6 teachers, the instructional coach, and 
the principal. Five of the participants were Caucasian females and one (a teacher) was a Hispanic female. 

The second school was located on the western edge of the urban area. It was a small school, serving approximately 
200 students in Kindergarten through grade 5. The school slowly decreased in size in the aftermath of school busing 
(racial balance actions). Class sizes were relatively small (e.g., 18 to 20 students). About half of the students were 
minorities (50% Caucasian, 45% black and 5% other). The CEC team at the school initially consisted of all teachers (11) 
and the principal, although the number of participants dropped between the first and third years to approximately 
6 teachers due to attrition. Ten of the participants were Caucasian (9 females and 1 male) and the principal was an 
African American female. 
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The third participating school was a Title I school located on the edge of the urban core area. The school served 
approximately 285 students of which 31% were of Hispanic descent and 40% were of African American descent. 
Additionally, 80% of the students came from economically disadvantaged households. The school was designated 
by the district as an English as a Second Language focus school. The CEC team at the school consisted of all the 4th 
grade teachers (3 total) and all of the 5th grade teachers (3 total), as well as the district appointed instructional coach 
for the school and the principal. All of the teachers and the instructional coach were Caucasian females. The principal 
was an African American female. 


RESULTS 


Making Sense ofTest Data 

One hurdle is the challenge of making sense of test data. Data from the project log support the notion that test data 
are often considered primarily as scores to be categorized into groups, such as exceeds standards, meets standards, 
approaches standards, and academic warning. While this categorization may be helpful for identifying students who 
need additional instruction, teachers need more information to effectively modify the learning opportunities for 
students. As one principal noted, "The assessment practices of [the teachers] have changed but now we need to 
focus on item analysis and how to better use the results to change instruction, then teachers can see the benefits of 
using test data." It is critical to understand what a test is actually designed to measure and to understand the extent to 
which a test actually measures what it purports to measure. One needs to think about such issues as the proportion 
of items on various subjects, the cognitive level of the items, and the content that is measured by a test. 

Tests typically include a variety of items designed to measure basic knowledge, problem solving skills, etc. To 
interpret test results and make sound instructional decisions based on results one must first understand the test. 
What makes the process so difficult when using outside tests, such as state accountability tests, is that changes are 
made every few years. This highlights the need for a continuous long-term data-based decision making process. Just 
when teachers think they understand the test system, it changes, and teachers need to adjust and re-interpret the 
results of a new test. 

In our schools, we spent a significant amount of time learning about the tests, the formats, the type of items, and what is 
claimed to be measured by the test.The school administrators played a key role in this work because test specification 
information was available to them and emphasized during school district meetings. Without leadership from the 
administrators, the teachers would not have had access to necessary, specific information about the tests. Shortly 
after our thorough review, the structure of the state mathematics test was changed. The proportion of geometry 
items on the state test was modified and the proportion of items at different cognitive levels was changed. At first 
this made the teachers in the CEC project feel frustrated because they had spent so much time understanding the 
test.The group lost some momentum because of the test changes, but eventually the teachers began to understand 
that change was inevitable. It actually helped them realize that the CEC process we were using needs to be viewed 
as a long-term continuous process because school environments are dynamic organizations whose context changes 
based on complex internal and external factors. 

Policy Changes 

Another external force for schools that came into play related to policy changes from the district evaluation and 
assessment office. The district was under increasing public pressure to help students achieve at higher and higher 
levels. In response, schools in the district were moving toward test preparation time periods to help students get 
ready for state testing. In many schools this translated to intense test preparation from January until the test date in 
April. During this preparation period students learned about item types, and teachers practiced basic skills with them 
in an effort to help students perform better on the test. 

In the schools we worked with, the CEC process was utilized to facilitate examining the alignment between test content, 
content standards, curriculum, and instruction. CEC teams studied the types of knowledge (procedural, conceptual, 
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factual, etc.), concepts, and skills students needed to successfully meet state content standards as measured through 
the state test. These studies were followed by an evaluation of the extent to which the curriculum and instructional 
practices teachers used aligned with the expectations of the standards. The focus on collaborative evaluation helped 
the administrators and teachers from our project conceptualize test preparation from a broader perspective. They 
were doing important work to align instruction with assessment rather than merely having students take repeated 
practice tests or complete targeted skill-based worksheets. 

When our teams finally felt comfortable with using test data, the district leadership mandated that schools stop test 
preparation for fear it would be viewed as "teaching towards the test." Concerns were raised about using assessment 
items in the classroom that were too similar to the items on the state test. The district leadership directed schools not 
to teach directly to the test but to focus on concepts related to the test. This was frustrating to the group because 
our CEC teams felt like we were finally making progress toward aligning instruction with assessment, but when the 
district sent a letter warning teachers about teaching to the test, the teams began to wonder about the extent to 
which we should align instruction with assessment. 

With assistance from the school administrators, we had to help teachers understand the difference between aligning 
instruction with assessment and actually teaching to the test. The two are quite different, but the district leadership 
did not make a clear distinction between them. This was puzzling to the teachers. They were using assessment 
information to diagnose and identify student weaknesses and then work with students to develop needed skills 
and understandings. As one teacher lamented, "What should I teach to? If they are going to test XYZ skills and 
judge my performance and school performance according to how well my kids do on XYZ, well of course I'm going 
to teach XYZ." Furthermore, because the state test is designed to be aligned with the state mathematics standards, 
teachers are not only teaching what they know will be on the test, but are also teaching toward the state mathematics 
standards. 

Fortunately, the CEC evaluation process teachers were using to engage in data-based decision making helped them 
deal with this issue. Schools try to help improve student achievement by working to align curriculum and instruction 
with state mathematics standards that are tested. Our teams focused on conceptual aspects of learning mathematics 
rather than having students learn how to answer specific test items. In some ways, the process helped teachers to 
focus on more comprehensive or global concepts students should know rather than on trivial details that are all too 
frequently included on a test. It also helped teachers think more about the state mathematics standards than the 
state mathematics test. 

Curriculum Changes 

A third factor that thwarted progress was when the mathematics curriculum was changed midstream in the process. 
Our CEC teams worked to understand how to use the curriculum to support instruction, what concepts and skills 
were measured through curriculum assessments, how to help student achieve, and then district policy mandated 
that the curriculum change. This change in curriculum was accompanied by a strict daily script of what should be 
taught each day. There was a district-wide pacing guide to tell teachers what they should be doing each day. 

Once again, having the collaborative evaluation communities' process in place allowed the schools to implement 
a curriculum analysis in the summer prior to using the new curriculum. Our teams identified alignment problems 
between the new curriculum and the state standards. On the one hand, the district leaders directed teachers to 
follow the pacing guide and stay on track with coverage of material, but because of our work, teachers knew about 
concepts and skills missing in the curriculum and wanted to do something to address the gaps. As a teacher described, 
"because of the CEC team's work I am more aware of Investigations [the district mathematics curriculum], we all know 
that Investigations has gaps and in the past we didn't have time to think about them.... [the CEC evaluation process] 
definitely helped show areas for improvement. We could ask ourselves, was it in Investigations? Was it missing from 
Investigations, or did we miss it instructional^?" 

The requirement to follow the pacing guide frustrated the teachers and school leaders on our teams. The CEC teams 
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had an understanding about the curriculum and teaching at a more specific school level than district administrators. 
They also knew their own students better than the district central office, and they knew what modifications needed 
to be made to the curriculum. The CEC evaluation process helped school leaders and teachers see the problems with 
the pacing guide. 



The process also gave the teachers more confidence in making modifications to the curriculum. In response to the 
district pacing guide, our teachers gathered evidence regarding gaps in the curriculum. They used this evidence 
to make the case with the district mathematics coordinator that they needed more freedom to address the gaps 
and to address areas where students were having difficulties. They were able to make a logical argument based 
on data with district leadership and to make the case that they needed freedom to help students achieve. The CEC 
evaluation process helped the principals and teachers make a strong case with data and helped shift the focus back 
to improving the curriculum for students. 


DISCUSSION 




Overall, these three challenges highlight external factors that can thwart progress when one engages in data-based 
decision making. The changes we faced regarding the state achievement test, the district constraints on preparing 
students for state testing, and the changes in the mathematics curriculum all highlight the need for those engaged 
in data-based decision making to understand that external factors will come into play. 

It is inevitable that change will occur because school systems are dynamic entities. The CEC data-based decision 
making process was a way to deal with and respond to changes. The data empowered school administrators and 
teachers to make professional and reasoned responses to the challenges they faced. The administrator's role was 
critical to this shift in approach. Not only was it necessary for the administrator to establish supportive conditions for 
the teachers to engage in the CEC process, but it was also important for the person in that role to share information 
that had typically been viewed as the purview of administrators rather than teachers. The focus on collaborative 
evaluation aided the data-based decision making of the CEC team by necessitating a shift in how collaborative work 
in the schools was viewed. The challenges helped administrators and teachers see the need for a continuous and 
collaborative evaluation process where teachers were actively involved in the analysis of data from a variety of sources. 
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When the process began, the school administrators believed internal challenges such as finding time, finding teachers 
who could do the work, and motivating teachers to engage in this process, would present the biggest challenges, but 
what was most surprising to administrators was that external factors became critical issues in the process. The CEC 
evaluation process became the most effective way for teachers to use data to make decisions and to overcome those 
external hurdles. The teachers became what Kowalski, Lasley, and Mahoney (2008) call "evidenced-based decision 
makers" (p. 256), and, as a result, they had a constructive way to deal with the challenge of helping students achieve. 
It is imperative that, as administrators engage teachers in data-based decision making, they also understand that 
outside barriers and hurdles are part of the process. Furthermore, the process itself is the vehicle for responding to 
external issues and for ultimately working to improve teaching and learning in such a way that all students have the 
opportunity to achieve. 
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